{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video Pipeline Details\n",
    "\n",
    "This notebook goes into detail about the stages of the video pipeline in the base overlay and is written for people who want to create and integrate their own video IP. For most regular input and output use cases the high level wrappers of `HDMIIn` and `HDMIOut` should be used.\n",
    "\n",
    "Both the input and output pipelines in the base overlay consist of four stages, an HDMI frontend, a colorspace converter, a pixel format converter, and the video DMA. For the input the stages are arranged Frontend -> Colorspace Converter -> Pixel Format -> VDMA with the order reversed for the output side. The aim of this notebook is to give you enough information to use each stage separately and be able to modify the pipeline for your own ends.\n",
    "\n",
    "Before exploring the pipeline we'll import the entire pynq.lib.video module where all classes relating to the pipelines live. We'll also load the base overlay to serve as an example.\n",
    "\n",
    "The following table shows the IP responsible for each stage in the base overlay which will be referenced throughout the rest of the notebook\n",
    "\n",
    "|Stage             | Input IP                               | Output IP                          |\n",
    "|------------------|:---------------------------------------|:-----------------------------------|\n",
    "|Frontend (Timing) |`video/hdmi_in/frontend/vtc_in`         |`video/hdmi_out/frontend/vtc_out`   |\n",
    "|Frontend (Other)  |`video/hdmi_in/frontend/axi_gpio_hdmiin`|`video/hdmi_out/frontend/axi_dynclk`|\n",
    "|Colour Space      |`video/hdmi_in/color_convert`           |`video/hdmi_out/color_convert`      |\n",
    "|Pixel Format      |`video/hdmi_in/pixel_pack`              |`video/hdmi_outpixel_unpack`        |\n",
    "|VDMA              |`video/axi_vdma`                        |`video/axi_vdam`                    |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pynq.overlays.base import BaseOverlay\n",
    "from pynq.lib.video import *\n",
    "\n",
    "base = BaseOverlay(\"base.bit\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HDMI Frontend\n",
    "\n",
    "The HDMI frontend modules wrap all of the clock and timing logic. The HDMI input frontend can be used independently from the rest of the pipeline by accessing its driver from the base overlay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdmiin_frontend = base.video.hdmi_in.frontend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the device will signal to the computer that a monitor is connected. Starting the frontend will wait attempt to detect the video mode, blocking until a lock can be achieved. Once the frontend is started the video mode will be available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VideoMode: width=1280 height=720 bpp=24"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdmiin_frontend.start()\n",
    "hdmiin_frontend.mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HDMI output frontend can be accessed in a similar way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdmiout_frontend = base.video.hdmi_out.frontend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the mode must be set prior to starting the output. In this case we are just going to use the same mode as the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdmiout_frontend.mode = hdmiin_frontend.mode\n",
    "hdmiout_frontend.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that nothing will be displayed on the screen as no video data is currently being send."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Colorspace conversion\n",
    "\n",
    "The colorspace converter operates on each pixel independently using a 3x4 matrix to transform the pixels. The converter is programmed with a list of twelve coefficients in the folling order:\n",
    "\n",
    "|     |in1 |in2 |in3 | 1  |\n",
    "|-----|----|----|----|----|\n",
    "|out1 |c1  |c2  |c3  |c10 |\n",
    "|out2 |c4  |c5  |c6  |c11 |\n",
    "|out3 |c7  |c8  |c9  |c12 |\n",
    "\n",
    "Each coefficient should be a floating point number between -2 and +2.\n",
    "\n",
    "The pixels to and from the HDMI frontends are in BGR order so a list of coefficients to convert from the input format to RGB would be:\n",
    "\n",
    "    [0, 0, 1,\n",
    "     0, 1, 0,\n",
    "     1, 0, 0,\n",
    "     0, 0, 0]\n",
    "\n",
    "reversing the order of the pixels and not adding any bias.\n",
    "\n",
    "The driver for the colorspace converters has a single property that contains the list of coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.0, -0.0, 1.0, -0.0, 1.0, -0.0, 1.0, -0.0, -0.0, -0.0, -0.0, -0.0]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colorspace_in = base.video.hdmi_in.color_convert\n",
    "colorspace_out = base.video.hdmi_out.color_convert\n",
    "\n",
    "bgr2rgb = [0, 0, 1,\n",
    "           0, 1, 0, \n",
    "           1, 0, 0,\n",
    "           0, 0, 0]\n",
    "\n",
    "colorspace_in.colorspace = bgr2rgb\n",
    "colorspace_out.colorspace = bgr2rgb\n",
    "\n",
    "colorspace_in.colorspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pixel format conversion\n",
    "\n",
    "The pixel format converters convert between the 24-bit signal used by the HDMI frontends and the colorspace converters to either an 8, 24, or 32 bit signal. 24-bit mode passes the input straight through, 32-bit pads the additional pixel with 0 and 8-bit mode selects the first channel in the pixel. This is exposed by a single property to set or get the number of bits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pixel_in = base.video.hdmi_in.pixel_pack\n",
    "pixel_out = base.video.hdmi_out.pixel_unpack\n",
    "\n",
    "pixel_in.bits_per_pixel = 8\n",
    "pixel_out.bits_per_pixel = 8\n",
    "\n",
    "pixel_in.bits_per_pixel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video DMA\n",
    "\n",
    "The final element in the pipeline is the video DMA which transfers video frames to and from memory. The VDMA consists of two channels, one for each direction which operate completely independently. To use a channel its mode must be set prior to start being called. After the DMA is started `readframe` and `writeframe` transfer frames. Frames are only transferred once with the call blocking if necessary. `asyncio` coroutines are available as `readframe_async` and `writeframe_async` which yield instead of blocking. A frame of the size of the output can be retrieved from the VDMA by calling `writechannel.newframe()`. This frame is not guaranteed to be initialised to blank so should be completely written before being handed back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputmode = hdmiin_frontend.mode\n",
    "framemode = VideoMode(inputmode.width, inputmode.height, 8)\n",
    "\n",
    "vdma = base.video.axi_vdma\n",
    "vdma.readchannel.mode = framemode\n",
    "vdma.readchannel.start()\n",
    "vdma.writechannel.mode = framemode\n",
    "vdma.writechannel.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "frame = vdma.readchannel.readframe()\n",
    "vdma.writechannel.writeframe(frame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, because we are only using 8 bits per pixel, only the red channel is read and displayed.\n",
    "\n",
    "The two channels can be tied together which will ensure that the input is always mirrored to the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vdma.readchannel.tie(vdma.writechannel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frame Ownership\n",
    "\n",
    "The VDMA driver has a strict method of frame ownership. Any frames returned by `readframe` or `newframe` are owned by the user and should be destroyed by the user when no longer needed by calling `frame.freebuffer()`. Frames handed back to the VDMA with `writeframe` are no longer owned by the user and should not be touched - the data may disappear at any time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up\n",
    "\n",
    "It is vital to stop the VDMA before reprogramming the bitstream otherwise the memory system of the chip can be placed into an undefined state. If the monitor does not power on when starting the VDMA this is the likely cause."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vdma.readchannel.stop()\n",
    "vdma.writechannel.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
